Valid parameters for predictive state representations

Predictive state representations (PSRs) represent the state of a dynamical system as a set of predictions about future events. The parameters of a PSR model consist of several matrices and vectors, but not all values for those parameters result in valid PSR models. Our work starts with a general definition of what it means to be a valid PSR model and derives necessary and sufficient constraints for the model parameters to constitute a valid PSR. These same constraints also define the set of valid state vectors for a given PSR model, which we prove to be a convex set. We also derive a set of simplified constraints on the PSR parameters, and we prove that any PSR model has an equivalent parameterization that satisfies those simplified constraints. Lastly, we demonstrate one simple application of our constraints: preventing overflow or underflow of the PSR state as it changes over time. Predictive state representations (PSRs) (Littman, Sutton, & Singh 2001) are a class of models that represent the state of a dynamical system as a set of predictions about future events. PSRs are capable of representing partially observable, stochastic dynamical systems, including any system that can be modeled by a finite partially observable Markov decision process (POMDP) (Singh, James, & Rudary 2004). There is evidence that predictive state is useful for generalization (Rafols et al. 2005) and helps to learn more accurate models than the state representation of a POMDP (Wolfe, James, & Singh 2005). Both POMDPs and PSRs have constraints on the matrices that constitute their parameters. For POMDPs, the constraints on the parameters are well known: particular subsets of the parameters must form stochastic vectors (i.e., vector elements are between 0.0 and 1.0 and sum to 1.0). In contrast, the nature of the constraints on PSR parameters has not been explicitly studied. Thus, current algorithms for learning PSR models from data disregard the fact that there are constraints on the PSR parameters (e.g., James & Singh (2004), Copyright c © 2009, authors listed above. All rights reserved. Wolfe, James, & Singh (2005), or Rosencrantz, Gordon, & Thrun (2004)). This can lead to invalid parameters and invalid state vectors that cause the PSR model to make predictions outside the range [0,1] of valid probabilities (Wolfe, James, & Singh 2005). This issue motivates the need to define the sets of valid parameters and state vectors so that future learning algorithms can find valid PSR parameters. This work provides a set of necessary and sufficient constraints on the parameters for a class of PSRs. In addition, these same constraints define the set of valid state vectors for a PSR. Thus, the constraints could be used both for ensuring that a learning algorithm produces valid PSR parameters and for checking that the state vectors used during the model’s operation are valid. 1 Background This work deals with models of discrete-time dynamical systems which have a set of discrete actions A and a set of discrete observations O. At each time step x, the agent chooses some action ax ∈ A to execute and then receives some observation ox ∈ O. A history is a possible sequence of actions and observations a1o1a2o2 . . . aτoτ from the beginning of time. A test is a sequence of possible future actions and observations aτ+1oτ+1 . . . aτ+koτ+k, where τ is the current time step. The prediction for a test t = aτ+1oτ+1 . . . aτ+koτ+k from a history h = a1o1 . . . aτoτ is defined as the probability of seeing the observations of t when the actions of t are taken from history h. Formally, this prediction is